Automatic appeal measurement system

ABSTRACT

An appeal estimation system for estimating a personal appeal of a candidate individual to an observer including a digital image capture device and a soft-copy display. The system further includes a data processing system configured to implement the steps of designating a set of proxy individuals; providing one or more digital images for each proxy individual; determining personal appeal values of each proxy individual to the observer; capturing one or more digital images of the candidate individual using the digital image capture device; determining image similarity metrics between the digital images for the candidate individual and each proxy individual; determining similarity values between the candidate individual and each proxy individual responsive to the determined image similarity metrics; estimating the personal appeal of the candidate individual to the observer by combining the personal appeal values for the proxy individuals; and displaying an indication of the estimated personal appeal on the soft-copy display.

CROSS-REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly assigned, co-pending U.S. patentapplication Ser. No. ______ (95865), entitled: “Automatic AppealMeasurement Method”, by Andrew C. Blose et al., which is incorporatedherein by reference.

FIELD OF THE INVENTION

This invention pertains to the field of image processing and dataanalysis, and more particularly to the automatic assessment of personalappeal.

BACKGROUND OF THE INVENTION

Appeal of one person to another is a subjective measure that varies withthe individuals. There are many applications when an assessment ofphysical attractiveness may be useful from an objective source. Examplesof such applications include online dating services; talent agencies;amusement providers; providers of professional services such asclinicians, estheticians and plastic surgeons; and employers looking tohire an actor, a performer, a model, or a subject for a demonstration.

Automated systems have been invented that provide an objective measureof physical attractiveness based on facial features provided from adigital image. For example, see the article “Assessing facial beautythrough proportion analysis by image processing and supervised learning”by Gunes et al. (Int. J. Human-Computer Studies, Vol. 64, pp. 1184-1199,2006). The objective measures provided by such models are based upon asingle universal estimate of appeal intended to approximate the averageappeal of that person on the population at large. However, facialfeatures alone are only part of physical attractiveness. Other physicalfeatures such as height, weight, and hair color and style can contributeto physical appeal. In addition, non-physical factors such as income,activities, level of education, personality, and social or politicalaffiliations may also influence the overall personal appeal oneindividual may have to another. Such factors may be reflected in aperson's style of dress, posture, and body language in a manner that istoo nuanced for computer algorithms to perceive, yet are obvious to thehuman observer. Previous systems suffer from the inability to adapt tolocal cultural norms and the context of a particular application sincethey offer a universal model of appeal and produce only a singleestimate of appeal for an individual.

What is needed is a system that allows for the automatic generation of ameasurement of appeal based on digital imagery and optionallypreferences learned from the user of the system.

SUMMARY OF THE INVENTION

The present invention represents an appeal estimation system forestimating a personal appeal of a candidate individual to an observercomprising:

a digital image capture device for capturing a digital image of thecandidate individual;

a soft-copy display;

a data processing system; and

a memory system communicatively connected to the data processing systemand storing instructions configured to cause the data processing systemto implement a method for estimating the personal appeal of thecandidate individual to the observer comprising:

-   -   a) designating a set of proxy individuals;    -   b) providing one or more digital images for each proxy        individual;    -   c) determining personal appeal values of each proxy individual        to the observer;    -   d) capturing one or more digital images of the candidate        individual using the digital image capture device;    -   e) determining image similarity metrics between the digital        images for the candidate individual and the digital images for        each proxy individual;    -   f) determining similarity values between the candidate        individual and each proxy individual responsive to the        determined image similarity metrics;    -   g) estimating the personal appeal of the candidate individual to        the observer by determining a weighted combination of the        personal appeal values for the proxy individuals, wherein the        weighted combination uses weighting coefficients that are        determined responsive to the similarity values between the        candidate individual and the corresponding proxy individual; and    -   h) displaying an indication of the estimated personal appeal on        the soft-copy display.

The present invention has the advantage that it provides a measure ofpersonal appeal corresponding to the preferences of a specific observer,rather than a general appeal metric based on preferences for apopulation of observers.

It has the additional advantage that a model of personal appeal for aspecific individual can be determined without requiring the observer toperform an extensive training process. This is accomplished by relatingthe preferences of the particular observer to those of a set of trainingobservers who have evaluated a large number of training individuals.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level diagram showing the components of a system forassessing the personal appeal a candidate individual to an observeraccording to an embodiment of the present invention;

FIG. 2 illustrates a flow-chart of a method for practicing the variousmethods of assessing the personal appeal a candidate individual to anobserver according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating a set of training observers andtraining individuals;

FIG. 4 is a diagram showing details relevant to the training of a proxyobserver model;

FIG. 5 is a flow diagram showing additional details of the estimatepersonal appeal step in FIG. 2;

FIG. 6 illustrates a data-centric view of the personal appeal estimationprocess of FIG. 2; and

FIG. 7 illustrates a portable appeal estimation device.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, some embodiments of the present inventionwill be described in terms that would ordinarily be implemented assoftware programs. Those skilled in the art will readily recognize thatthe equivalent of such software may also be constructed in hardware.Because image manipulation algorithms and systems are well known, thepresent description will be directed in particular to algorithms andsystems forming part of, or cooperating more directly with, the methodin accordance with the present invention. Other aspects of suchalgorithms and systems, together with hardware and software forproducing and otherwise processing the image signals involved therewith,not specifically shown or described herein may be selected from suchsystems, algorithms, components, and elements known in the art. Giventhe system as described according to the invention in the following,software not specifically shown, suggested, or described herein that isuseful for implementation of the invention is conventional and withinthe ordinary skill in such arts.

The invention is inclusive of combinations of the embodiments describedherein. References to “a particular embodiment” and the like refer tofeatures that are present in at least one embodiment of the invention.Separate references to “an embodiment” or “particular embodiments” orthe like do not necessarily refer to the same embodiment or embodiments;however, such embodiments are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art. The useof singular or plural in referring to the “method” or “methods” and thelike is not limiting. It should be noted that, unless otherwiseexplicitly noted or required by context, the word “or” is used in thisdisclosure in a non-exclusive sense.

The phrase, “digital image,” as used herein, refers to any digitalimage, such as a digital still image or a digital video.

FIG. 1 is a high-level diagram showing the components of a system forautomatically assessing the personal appeal of candidate individuals toan observer according to an embodiment of the present invention. Thesystem includes a data processing system 110, a peripheral system 120, auser interface system 130, and a processor-accessible memory 140. Theperipheral system 120, the user interface system 130 and theprocessor-accessible memory 140 are communicatively connected to thedata processing system 110.

The data processing system 110 includes one or more data processingdevices that implement the processes of the various embodiments of thepresent invention, including the example processes described herein. Thephrases “data processing device” or “data processor” are intended toinclude any data processing device, such as a central processing unit(“CPU”), a desktop computer, a laptop computer, a mainframe computer, apersonal digital assistant, a Blackberry™, a digital camera, cellularphone, or any other device for processing data, managing data, orhandling data, whether implemented with electrical, magnetic, optical,biological components, or otherwise.

The processor-accessible memory 140 includes one or moreprocessor-accessible memories configured to store information, includingthe information needed to execute the processes of the variousembodiments of the present invention, including the example processesdescribed herein. The processor-accessible memory 140 may be adistributed processor-accessible memory system including multipleprocessor-accessible memories communicatively connected to the dataprocessing system 110 via a plurality of computers or devices. On theother hand, the processor-accessible memory 140 need not be adistributed processor-accessible memory system and, consequently, mayinclude one or more processor-accessible memories located within asingle data processor or device.

The phrase “processor-accessible memory” is intended to include anyprocessor-accessible data storage device, whether volatile ornonvolatile, electronic, magnetic, optical, or otherwise, including butnot limited to, registers, floppy disks, hard disks, Compact Discs,DVDs, flash memories, ROMs, and RAMs.

The phrase “communicatively connected” is intended to include any typeof connection, whether wired or wireless, between devices, dataprocessors, or programs in which data may be communicated. The phrase“communicatively connected” is intended to include a connection betweendevices or programs within a single data processor, a connection betweendevices or programs located in different data processors, and aconnection between devices not located in data processors at all. Inthis regard, although the processor-accessible memory 140 is shownseparately from the data processing system 110, one skilled in the artwill appreciate that the processor-accessible memory 140 may be storedcompletely or partially within the data processing system 110. Furtherin this regard, although the peripheral system 120 and the userinterface system 130 are shown separately from the data processingsystem 110, one skilled in the art will appreciate that one or both ofsuch systems may be stored completely or partially within the dataprocessing system 110.

The peripheral system 120 may include one or more devices configured toprovide digital content records to the data processing system 110. Forexample, the peripheral system 120 may include digital still cameras,digital video cameras, cellular phones, or other data processors. Thedata processing system 110, upon receipt of digital content records froma device in the peripheral system 120, may store such digital contentrecords in the processor-accessible memory 140.

The user interface system 130 may include a mouse, a keyboard, anothercomputer, or any device or combination of devices from which data isinput to the data processing system 110. In this regard, although theperipheral system 120 is shown separately from the user interface system130, the peripheral system 120 may be included as part of the userinterface system 130.

The user interface system 130 also may include a display device, aprocessor-accessible memory, or any device or combination of devices towhich data is output by the data processing system 110. In this regard,if the user interface system 130 includes a processor-accessible memory,such memory may be part of the processor-accessible memory 140 eventhough the user interface system 130 and the processor-accessible memory140 are shown separately in FIG. 1.

In the following discussions, the term candidate individual will be usedto refer to an individual whose personal appeal is being estimated orestablished. Generally, candidate individuals are individuals that arenew to the system for which personal appeal values have not beenpreviously established or estimated. The personal appeal of thecandidate individual is expressed with respect to a person who will bereferred to as the observer. The term client observer refers to anobserver on whose behalf the attractiveness value of a candidateindividual is being estimated. The candidate individual can also bereferred to as a target candidate or simply as a candidate, and a clientobserver may be referred to as an observer or a user. The terms trainingobserver and training individual will be used to respectively refer toobservers and individuals employed to provide appeal values on whichfuture estimates will be based. The terms attraction, attractiveness,personal appeal and appeal are considered to be synonymous in thesediscussions.

The present invention will now be described with reference to FIG. 2.There is shown a flow diagram illustrating a method for estimating thepersonal appeal (attractiveness) of a candidate individual to a clientobserver. In order to estimate the personal appeal of a candidateindividual to a user, the system must first be trained relative to thepreferences of the observer. First a collect training data step 210 isused to collect personal appeal data for a set of training observers. Inthis step, the training observers provide personal appeal values foreach member of a set of training individuals. The personal appeal valuesare determined, at least in part, by viewing photographs (e.g., digitalimages) of the training individuals to evaluate their physicalattractiveness, although other factors may also be considered.Generally, the client observer will not be one of the trainingobservers.

A train proxy observer model step 220 is next used to enroll a new user(the client observer) in the system. The enrollment process involveslearning the preferences of the new user in order to construct a proxyobserver model that emulates these preferences by correlating the userpreferences with the preferences of a set of proxy observers. Note thatthe enrollment process does not have to be associated with a time ofenrollment of an observer in a system. In general, it can be performedat any time to determine a new or updated evaluation of observerpreferences.

In acquire candidate individual information step 230, informationdescribing the candidate individual is acquired. In a preferredembodiment, the acquired information includes at least one digital image(i.e., a photograph or video) of the candidate individual. Additionalinformation may also be acquired such as height, weight, age, education,income and interests.

Next, an estimate personal appeal step 240 is used to estimate thepersonal appeal of the candidate individual with respect to the clientobserver. This step includes using a processor to determine similarityvalues between the candidate individuals and a set of proxy individuals.The similarity values are based, at least in part, on image similaritymetrics determined between a digital image for the candidate individualsand digital images for a set of proxy individuals. The similarity valuesare used to determine weighting coefficients that are used for aweighted combination of personal appeal values for the proxyindividuals. In a preferred embodiment, the personal appeal values forat least a subset of the proxy individuals are determined using theproxy observer model determined in the train proxy observer model step220. In some embodiments, the personal appeal values for some (or all)of the proxy individuals are determined by having the client observerevaluate the personal appeal of the proxy individuals.

A store estimated personal appeal step 250 is used to store theresulting estimated personal appeal of the candidate individual to theclient observer in a processor-accessible memory. The estimated personalappeal can also be reported to the client observer, for example bydisplaying the personal appeal value on a user interface.

In some embodiments, the client observer is given the opportunity toconfirm or correct the estimated personal appeal value. If so, thecorrected personal appeal value is stored in the processor-accessiblememory for future reference. In some embodiments, the determinedpersonal appeal value for the candidate individual can be used in theprocess of estimating the personal appeal of future candidateindividuals. This is most appropriate for cases where the clientobserver has confirmed or corrected the estimated personal appeal valuesince these personal appeal values should represent more accuratetraining data. One way that the determined personal appeal value can beused in the process of estimating the personal appeal of futurecandidate individuals is to add the candidate individual to the set ofproxy individuals.

FIG. 3 illustrates additional details relevant to the collect trainingdata step 210. In this step, training data is collected for a set oftraining observers 310 and a set of training individuals 320. Theelements O_(T,k) represent the k^(th) training observer 310. Theelements C_(T,j) represent the j^(th) training individual 320. The linesjoining the training observers and training individuals represent thecorresponding appeal values 330 of the training individuals 320 to thetraining observers 310. The appeal value of the j^(th) trainingindividual to the k^(th) training observer will be referred to asA(C_(T,j), O_(T,k)). This appeal values 330 are obtained by displayinginformation about a training individual 320 to a training observer 310and recording the appeal value 330 provided by the training observer310. In a preferred embodiment, the displayed information about atraining individual includes a digital image (e.g., a digital stillimage or a digital video) of the training individual. The displayedinformation can also include other types of information such as age,height, weight, hair color, hair style, income, activities, level ofeducation, personality, social affiliations and political affiliations.This set of training data forms the basis from which future estimates ofpersonal appeal will be derived.

The set of training individuals 320 and training observers 310 shouldpreferably be large enough to encompass much of the inherent diversityin each population. The diversity of the population of trainingindividuals 320 may result in a large training task for each of thetraining observers 310. Therefore, it may be preferable to limit in thetraining task by forming a subset of the training individual populationby sampling the training individual population for each trainingobserver 310. In the preferred embodiment, the sampling is non-uniformand favors the selection of training individuals 320 with similardemographic background. For instance, the probability of the selectionof a training individual 320 drops as the difference between the ages ofthe training individual 320 and training observer 310 increases.

FIG. 4 is a diagram illustrating additional details relevant to thetrain proxy observer model step 220. When a new observer 410 is added tothe system, the preferences of the new observer 410 are assessed, and aproxy observer model is formed that can be used to predict thosepreferences by correlating the new observer preferences with thepreferences of a set of proxy observers. In a preferred embodiment, thenew observer 410 is presented with, a subset of the training individualpopulation called an enrollment subset 420 and asked to providecorresponding new observer appeal values 430. The new observer appealvalue of the j^(th) training individual to the new observer will bereferred to as B(C_(T,j), O).

The enrollment subset 420 may be far smaller than the total populationof training individuals 320. In a preferred embodiment, the enrollmentsubset 420 is selected in such a way as to cover much of the diversityof the population of training individuals 320. In other embodiments, theenrollment subset 420 can be selected to emphasize a particulardemographic range of interest. For example, the enrollment subset 420can be selected to include only training individuals 320 within aspecific age range according to preferences expressed by the observer410.

Once the new observer has provided new observer appeal values 430 foreach member of the enrollment subset 420, a subset of training observerscalled proxy observers 440 is selected. Proxy observers 440 are a set oftraining observers 310 whose appeal values 330 for the enrollment subsetcan be combined in such a way as to approximate the new observer appealvalues 430 associated with the new observer 410. In a preferredembodiment, the proxy observers 440 are chosen such that a linearcombination of their appeal values 330 closely approximates the newobserver appeal values 430 associated with the new observer. In equationform, this can be expressed as:

$\begin{matrix}{{B\left( {C_{T,j},O} \right)} \approx {\sum\limits_{k = 1}^{N}{X_{k}{A\left( {C_{T,j},O_{T,k}} \right)}}}} & (1)\end{matrix}$

where N is the number of proxy observers 440, and X_(k) is a weightingcoefficient for the k^(th) proxy observer. The selected proxy observers440 and respective set of weighting coefficients constitute a proxyobserver model that can be used to approximate appeal values for theremaining training individuals 320 that have not been evaluated by thenew observer 410.

Those skilled in the art will recognize that there are a variety ofmethods that can be used to select the proxy observers 440 and calculatethe corresponding weighting coefficients. In one embodiment, the entirepopulation of training observers 310 are used as proxy observers 440.The corresponding weighting coefficients can then be determined usingstandard least-squares regression. More formally, the least squareregression process can be expressed as:

min∥AX−B∥ ₂ ²  (2)

where ∥•∥₂ is the 2-norm operator, A is an M×N matrix of appeal values330 between training observers 310 in the set of proxy observers 440 andtraining individuals 320 in the enrollment subset 420:

$\begin{matrix}{A = \begin{bmatrix}{{A\left( {C_{T,1},O_{T,1}} \right)},{A\left( {C_{T,1},O_{T,2}} \right)},\ldots \mspace{14mu},{A\left( {C_{T,1},O_{T,N}} \right)}} \\{{A\left( {C_{T,2},O_{T,1}} \right)},{A\left( {C_{T,2},O_{T,2}} \right)},\ldots \mspace{14mu},{A\left( {C_{T,2},O_{T,n}} \right)}} \\\vdots \\{{A\left( {C_{T,M},O_{T,1}} \right)},{A\left( {C_{T,M},O_{T,2}} \right)},\ldots \mspace{14mu},{A\left( {C_{T,M},O_{T,N}} \right)}}\end{bmatrix}} & (3)\end{matrix}$

M being the number of training individuals 320 in the enrollment subset420, B is an M-length vector of new observer appeal values 430 providedby the new user for the training individuals 320 in the enrollmentsubset 420:

B=[B(C _(T,1) ,O),B(C _(T,2) ,O), . . . , B(C _(T,M) ,O)]^(T)  (4)

[•]^(T) being the transpose operator, and X is an N-length vector ofweighting coefficients associated with the proxy observers 440 that aredetermined to minimize the prediction error:

X=[X ₁ ,X ₂ , . . . , X _(N)]^(T)  (5)

It can be seen that only a subset of the appeal values 330 in FIG. 4 areinvolved in these calculations. This subset is shown using solid lines,while the unused appeal values 330 are shown using dashed lines. Oncethe proxy observer model has been formed, it can be used to predict thepersonal appeal of any of the training individuals 320 to the newobserver 410 using Eq. (1), even including the training individuals 320not in the enrollment subset 420.

Those skilled in the art will also recognize that a smaller set of proxyobservers 440 can be selected and corresponding weighting coefficientmay be obtained using well-known optimization techniques such as thosedescribed by Boyd and Vandenberghe in Chapter 6 of the book “ConvexOptimization” (Cambridge University Press, Cambridge, 2004). In someembodiments, the basis pursuit technique described in section 6.5.4 ofthe above mentioned book, is used to determine a minimum number of proxyobservers needed to provide an adequate estimate of an observer'spersonal preferences.

In the example shown in FIG. 4, the number of training individuals 320in the enrollment subset 420 is M=3 and the number of proxy observers440 is N=2, although preferably a larger enrollment subset 420 and alarger number of proxy observers 440 would be used. In practice, thesize of the enrollment subset 420 will be limited by the number ofevaluations that the new observer 410 is willing to make. If the numberis too large, the enrollment process will become too cumbersome for thenew observer 410 to tolerate. The number of proxy observers 440 can beas large as necessary to provide a good fit to the new observer appealvalues 430, without over-fitting the random variability in the data.

The linear combination of appeal values 330 given in Eq. (1) is aspecific example of a more general case where the new observer appealvalues 430 are expressed as a weighted combination of the appeal values330 for the proxy observers 440, wherein the weighted combination usesweighting coefficients determined responsive to the new observer appealvalues 430 and the appeal values 330 for the proxy observers 440 in theset of training observers 310. In other embodiments, the weightedcombination can include additional terms beyond the simple linearcombination terms. For example, the weighted combination can includehigher-order terms such as quadratic terms and cross-terms.

FIG. 5 illustrates additional details relevant to the estimate personalappeal step 240, which estimates the personal appeal of a candidateindividual to a particular observer. The appeal estimation processbegins with an identify proxy individuals step 510 which identifies aset of individuals that will act as proxies for a candidate individual.The set of proxy individuals can be composed of a combination oftraining individuals 320 and previous individuals for which theparticular observer has provided actual personal appeal values. In apreferred embodiment, the proxy individuals are selected by measuringthe similarity of the candidate individuals to each potential proxyindividuals. The set of proxy individuals may be limited to thosepotential individuals having a degree of similarity that surpasses apredefined threshold, or by limiting the number of proxy individuals toa specified number of individuals having the highest degree ofsimilarity.

Those skilled in the art will recognize that many similaritymeasurements and features may be used within the scope of thisinvention. In a preferred embodiment, the degree of similarity isdetermined responsive to an image similarity metric which represents thesimilarity between digital images of the candidate individual and theproxy individuals. In some embodiments, the image similarity metric isdetermined using a facial similarity metric which represents thesimilarity of the faces of the candidate individual and the proxyindividual. Any algorithm for determining facial similarity metricsknown in the art can be used in accordance with the present invention.For example, a survey of applicable methods for determining facialsimilarity metrics is described by W. Zhao et al. in the article “FaceRecognition: A Literature Survey” (ACM Computing Surveys, Vol. 35, pp.399-458, 2003).

In a preferred embodiment, a facial similarity metric based upon aunified Principle Component Analysis (PCA) and Linear DiscriminantAnalysis (LDA) can be employed. Using this approach, each face can berepresented using a principle component model that expresses both theshape and texture of the face. An example of this is the ActiveAppearance Model (AAM) described by Cootes et al. in the article “ActiveAppearance Models” (Proc. 5th European Conference on Computer Vision,Vol. II, pp. 484-498, 1998). The similarity between any two faces can beestablished by comparing the respective eigenvalues. A coordinatetransformation for the eigenvalues can be determined using LDA so as tomaximize the distance between faces which depict different people whileminimizing the distance between faces which depict the same person. Thedistance between the representations of two faces in this transformedspace is therefore a measure of the dissimilarity between the two faces,where larger distances correspond to larger differences.

The dissimilarity distance value can then be used to determine thefacial similarity metric by applying Bayesian analysis as described inthe aforementioned article by Zhao et al. Alternately, the dissimilaritydistance value can be mapped through an appropriate nonlineartransformation, such as a sigmoid function of the form:

$\begin{matrix}{S_{f} = \frac{1}{1 + ^{- {m{({h - D_{f}})}}}}} & (6)\end{matrix}$

where S_(f) is the facial similarity metric, D_(f) is the facialdissimilarity distance value, h is the inflection point of the sigmoid,and m is the slope of the sigmoid at the inflection point.

For the case where more than one digital image of a particularindividual is available, image similarity metrics can be computed foreach of the digital images. In one embodiment, the image similaritymetric with the highest value for a pair of individuals is selected. Inother embodiments, the image similarity metrics computed for thedifferent digital images can be combined (e.g., by averaging them).

A personal appeal known test 520 is used to determine whether the appealof each proxy individual to the observer is known. Appeal values will beavailable for training individuals 320 in the enrollment subset 420, aswell as for previous candidate individuals for which the observer hasprovided actual appeal values. If the personal appeal value of aparticular proxy individual to the observer is known, a retrievepersonal appeal step 530 is used to retrieve the personal appeal valuefor use in subsequent calculations.

If the personal appeal value of a particular proxy individual to theobserver is not known, a determine personal appeal using proxy observermodel step 550 is used to estimate the personal appeal of the particularproxy individual to the observer. This is accomplished using the proxyobserver model determined in the train proxy observer model step 220(FIG. 2). As described earlier, the proxy observer model estimates thepersonal appeal of an individual by determining a weighted combinationof personal appeal values of the individual to the proxy observers(e.g., using Eq. (1)).

In determine personal appeal of candidate individual step 560, apersonal appeal value of the candidate individual to the observer isestimated by calculating a weighted combination of the personal appealvalues for proxy individuals. The weighted combination combines thepersonal appeal values using weighting coefficients that are created bynormalizing the similarity scores for each proxy individual.

The personal appeal values of the proxy individuals, together withcorresponding information about the proxy individuals constitutes apersonal appeal model for the observer. In a preferred embodiment theinformation about the proxy individuals includes digital images of theproxy individuals, as well as other information that can be used todetermine similarity values between a candidate individual and the proxyindividuals.

In a preferred embodiment, the similarity scores are determinedresponsive to the image similarity metric which represents thesimilarity between digital images of the candidate individual and theproxy individuals. As discussed earlier, the image similarity metric ispreferably a facial similarity metric which represents the similarity ofthe faces of the candidate individual and the proxy individual. In someembodiments, the similarity score can also incorporate other factorssuch as similarity in age, height, weight, hair color, hair style,income, activities, level of education, personality, cultural heritage,religion, social affiliations and political affiliations.

In some embodiments, a similarity score incorporating other factorsbesides facial similarity can be determined by computing a combineddistance metric that incorporates the facial dissimilarity distancevalue as well as other distance terms corresponding to the otherfactors:

$\begin{matrix}{D = \sqrt{{c_{f}D_{f}^{2}} + {\sum\limits_{i}{c_{i}D_{i}^{2}}}}} & (7)\end{matrix}$

where D is the combined distance metric, D_(f) is the facialdissimilarity distance value, D_(i), is a dissimilarity distance valuefor the i^(th) factor, and C_(f) and C_(i) are weighting factors for thecorresponding distance values.

For some of the other factors, such as age, height and weight, which arenumerical in nature the dissimilarity distance value can simply be thedifference between the numerical values (e.g., the difference in theages). Alternately, a non-linear transformation can be applied to thedifference between the numerical values (e.g., to place added emphasizeon large age differences). For other factors, such as hair color or hairstyle, which do not have inherent numerical values, various schemes canbe used to define dissimilarity distance values. For example, a binarysystem can be used where a matching attribute is assigned a value of “0”while a non-matching attribute is assigned a value of “1.”

A combined similarity score can then be determined from the combineddistance metric by applying an appropriate nonlinear transformation,such as a sigmoid function shown in Eq. (6). Alternately, a combinedsimilarity metric can be determined by applying Bayesian analysis asdescribed in the aforementioned article by Zhao et al.

In alternate embodiments, other methods can be used to determine acombined similarity score that combines a facial similarity value withvarious other features. For example, U.S. Patent Application Publication2008/0080745 teaches several methods for determining a similarity scorebased on a facial similarity value.

In a preferred embodiment, the determination of the personal appealvalue of the candidate individual to the observer is estimated bycalculating a weighted summation of the personal appeal values for proxyindividuals:

$\begin{matrix}{{A\left( {C,O} \right)} \approx {\sum\limits_{j = 1}^{M_{p}}{W_{j}{A\left( {C_{P,j},O} \right)}}}} & (7)\end{matrix}$

where C is the candidate individual, O is the observer, C_(P,j) is thej^(th) proxy individual, M_(p) is the number of proxy individuals, W_(j)is a weighting coefficient for the j^(th) proxy individual, A(C_(P,j),O) is the personal appeal value of the j^(th) proxy individual to theobserver and A(C, O) is the estimated personal appeal value of thecandidate individual to the observer. It will be obvious to one skilledin the art that other types of weighted combinations can be used toestimate the personal appeal value of the candidate individual inaccordance with the present invention. For example, the weightedcombination can include nonlinear terms such as quadratic terms.

In a preferred embodiment, the weighting coefficient W_(j) for thej^(th) proxy individual can be determined from the similarity scoresusing the following equation:

$\begin{matrix}{W_{j} = \frac{S\left( {C,C_{P,j}} \right)}{\sum\limits_{k = 1}^{M_{p}}{S\left( {C,C_{P,k}} \right)}}} & (8)\end{matrix}$

where S(C, C_(P,j)) is the similarity score between the candidateindividual C and the j^(th) proxy individual C_(P,j). It can be seenthat the effect of this equation is that the weighting coefficients aredetermined by normalizing the similarity scores by the sum of thesimilarity scores. Thus the sum of the weighting coefficients for theproxy individuals will be 1.0. The weighting coefficients for the proxyindividuals that are most similar to the candidate individual will behighest, and the weighting coefficients for the proxy individuals thatare least similar to the candidate individual will be lowest.

FIG. 6 illustrates a data-centric view of the appeal estimation processdescribed by FIG. 2. The goal is to estimate the appeal of a candidateindividual 690 to an observer 610. A set of proxy individuals 620 knownto the system is selected to act as a proxy for the candidate individual690. Some of the proxy individuals 620 may be chosen from a set ofpreviously observed individuals 680 for which actual appeal values 650are known, and some may be chosen from the training individuals 320. Itshould be noted that the training individuals 320 selected for use asproxy individuals 620 is independent of the selection of the trainingindividuals 320 for use in the enrollment subset 420 (FIG. 4).

In some embodiments, the set of proxy individuals 620 includes onlypreviously observed individuals 680 and no training individuals 320. Inorder for this approach to produce high quality results, it is necessarythat a fairly large number of previously observed individuals 680 shouldbe evaluated by the observer 610. This implies that the enrollmentprocess would need to be more extensive than would be necessary when theset of proxy individuals 620 also contains training individuals 320. Insome embodiments, the set of proxy individuals 620 includes onlytraining individuals 320 and no previously observed individuals 680.

The appeal values 650 for each of the proxy individuals 620 chosen frompreviously observed individuals 680 can be used without respect to theproxy observers 440. The appeal values 650 may have been determined bythe observer 610 directly evaluating the previously observed individuals680 during the enrollment process, or at some other time. In someembodiments, the client observer 610 is given the opportunity to confirmor correct the estimated personal appeal value of a particularindividual. If so, the set of previously observed individuals 680 can beupdated to include the particular individual.

Appeal values 650 for each of the proxy individuals 620 chosen from theset of training individuals 320 that were not directly evaluated by theobserver 610 are estimated using the proxy observer model describedearlier. The proxy observer model estimates the personal appeal value ofan individual by computing a weighted combination of the appeal values330 between a particular training individual 320 and the proxy observers440 established for the observer 610. In a preferred embodiment, theweighted combination is performed using the weighted summation given inEq. (1). The weighted summation uses weighting coefficients 660 (X_(k))that were established when the proxy observers were chosen at enrollmenttime. The appeal values 330 that are involved in the calculation of thepersonal appeal of the proxy individuals are shown using solid lines inFIG. 6. The appeal values 330 that are not involved in thesecalculations are shown using dashed lines.

Finally, in a preferred embodiment, the personal appeal value of thecandidate individual 690 is estimated by computing a weighted sum of thepersonal appeal values of the proxy individuals 620. As describedearlier with respect to Eq. (7), the weighted sum is calculated usingweighting coefficients 670 (W_(j)) that are determined by normalizingsimilarity values between the proxy individuals 620 to the candidateindividual 690 as described in Eq. (8).

While the preferred embodiment of this invention applies to the personalappeal of one human to another, those skilled in the art will readilyrecognize that the invention can be exercised to estimate the appeal ofany type of “individual,” subject to the availability of a validsimilarity measurement that accurately estimates the similarity betweenindividuals. For instance, if one were to provide a metric to estimatethe similarity between automobiles based upon features such as color,size, horsepower, body style, etc. one could exercise the method of thisinvention to estimate the appeal of a candidate car to an observer. Anexample of such a metric could be to classify each automobile by bodystyle: coupe, sedan, SUV, crossover, van, pickup, etc. Similarity ofcontinuous values such as color, size, and horsepower could bedetermined by comparing the difference in their respective values to athreshold. Based on these features, a weighted Hamming distance betweenautomobiles could be employed to provide an overall similaritymeasurement.

The invention as so far described is single directional, in that thepersonal appeal of the candidate individual to the observer has beenestimated without any consideration of the personal appeal of theobserver to the candidate individual. However, it will be obvious to oneskilled in the art that the present invention may be exercised in amanner to determine a mutual appeal. In this configuration, anindividual may provide information to the system with the goal ofidentifying other individuals that they would find appealing and who arelikely to find the individual mutually appealing. One means forcomputing a mutual appeal value would be to compute the appeal of thefirst individual to the second individual using the above describedmethod. Likewise, the appeal of the second individual to the firstindividual can be determined in the same fashion. The two appeal valuescan then be combined in some appropriate manner such as by forming thesum or product of the appeal values. The system can then report themutual appeal value in addition to, or instead of the single-directionalappeal value.

Another application of the present invention is to provide observerswith information about what they could do to make themselves moreappealing to others. For example, a set of target individuals can beidentified that the observer finds to be appealing. Appeal models foreach of these target individuals can then be used to identify otherindividuals that the target individuals find to be appealing.Information about this set of other individuals (e.g., photographs,activities, social affiliations and political affiliations) can beshared with the observer to provide insight about ways for the observerto optimize his or her appeal to the set of target individuals. Inanother embodiment, observers can use image editing software to modifytheir personal appearance in order to evaluate how it would affect theirappeal to the set of target individuals. For example, the observer canevaluate attributes such as hair style, facial hair style, eyeglasses,dental work, facial jewelry, clothing style and facial expression wouldaffect their level of appeal.

Another application of the present invention is to create a model ofappeal for a population of observers rather than for a single observer.This can be done by combining appeal values for a set of observers thatrepresent the desired population. For instance, an advertiser may wishto target males between the ages of 39 and 55, from a particulargeographic region, having a desired income range. a set of observers canbe identified that satisfy a set demographic criteria specified by theadvertiser, and an overall appeal value can be determined by combiningindividual appeal values corresponding to each observer. Using thismodel, they can identify candidate individuals with the highest appealto this chosen population.

A computer program product can include one or more storage medium, forexample; magnetic storage media such as magnetic disk (such as a floppydisk) or magnetic tape; optical storage media such as optical disk,optical tape, or machine readable bar code; solid-state electronicstorage devices such as random access memory (RAM), or read-only memory(ROM); or any other physical device or media employed to store acomputer program having instructions for controlling one or morecomputers to practice the method according to the present invention.

According to some embodiments of the present invention, theabove-described method for estimating the personal appeal of a candidateindividual is included in a portable appeal estimation device. FIG. 7shows an example of such a portable appeal estimation device 700. Inthis embodiment, the portable appeal estimation device 700 includes adigital image capture device 705, such as a digital camera, which can beused by an observer 720 to capture digital images of a candidateindividual 730. The appeal estimation device 700 also includes asoft-copy display 710, which can, for example, be located on the back ofthe appeal estimation device 700. In some embodiments, the appealestimation device 700 is mobile computational device such as a cellulartelephone device or a tablet computer.

In a preferred embodiment, the appeal estimation device 700 alsoincludes a data processing system (not shown), a personal appeal modelmemory system (not shown) communicatively connected to the dataprocessing system and storing information about a set of proxyindividuals including digital images of each proxy individual andpersonal appeal values of each proxy individual to the observer, and aprogram memory system (not shown) communicatively connected to the dataprocessing system and storing instructions configured to cause the dataprocessing system to determine an estimated personal appeal of thecandidate individual to the observer using the above-described method.An indication of the resulting estimated personal appeal is displayed tothe observer 720 on the soft-copy display 710.

In some embodiments, the indication of the personal appeal can be anumerical value (e.g., a number between 1 and 10). In other embodiments,the indication of the personal appeal can be provided in some other formsuch as a star rating (e.g., 1 to 5 stars) or a bar graph where thelength of the bar is an indication of the estimated personal appeal. Itwill be obvious to those skilled in the art that there are a widevariety of ways that can be used to represent the estimated personalappeal value.

In some embodiments, the process of training a personal appeal model(e.g., the process of determining personal appeal values of a set ofproxy individuals to the observer) can be carried out by performing theabove-described enrollment process using software stored in the memorysystem within the appeal estimation device 700 and executed using thedata processing system within the appeal estimation device 700.

In other embodiments, the personal appeal model is trained using aprocess which is executed using external system components such as aremote computer. The resulting personal appeal model can then be loadedinto the appeal estimation device 700, or can be retained on the remotecomputer. Retaining the personal appeal model on the remote computer canbe a desirable alternative when the appeal estimation device 700 doesnot have the computational power required to process a captured digitalimage using the personal appeal model.

In some embodiments, the process of analyzing a captured digital imagedetermine an estimated personal appeal can be carried out using softwarestored in the memory system within the appeal estimation device 700 andexecuted using the data processing system within the appeal estimationdevice 700. In this configuration, the appeal estimation device 700 is astand-alone device for determining personal appeal values.

In alternate embodiments, the captured digital image is transmitted to aremote computing system for processing using a remote data processor.For example, the appeal estimation device 700 can be a cellulartelephone such that captured digital image can be transmitted using acellular telephone connection. Alternately, the captured digital imagecan be transmitted over a wired connection or over some other form ofwireless connection, such as a WiFi connection or a Bluetoothconnection. The estimated personal appeal value determined on the remotecomputing system is then transmitted back to the appeal estimationdevice 700 for display on the soft-copy display 710.

Portable appeal estimation devices 700 can be useful for individualusers who wish to evaluate the personal appeal of individuals they mayencounter. In this case, the appeal estimate device 700 can beconfigured to determine the personal appeal relative to the preferencesof the individual user. Portable appeal estimation devices 700 can alsobe useful for applications such as talent agencies evaluating potentialclients; providers of professional services such as clinicians,estheticians and plastic surgeons; and employers looking to hire anactor, a performer, a model, or a subject for a demonstration. In thesecase, the appeal estimation device 700 can be configured to determinethe appeal relative to a specified demographic. In one embodiment, amenu can be provided on a user interface of the estimation device 700that allows the user to choose between a number of predefineddemographic segments.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   -   110 Data processing system    -   120 Peripheral system    -   130 User interface system    -   140 Processor-accessible memory system    -   210 Collect training data step    -   220 Train proxy observer model step    -   230 Acquire candidate individual information step    -   240 Estimate personal appeal step    -   250 Store estimated personal appeal step    -   310 Training observer    -   320 Training individual    -   330 Appeal value    -   410 New observer    -   420 Enrollment subset    -   430 New observer appeal value    -   440 Proxy observers    -   510 Identify proxy individuals step    -   520 Personal appeal known test    -   530 Retrieve personal appeal step    -   550 Determine personal appeal using proxy observer model step    -   560 Determine personal appeal of candidate individual step    -   610 Observer    -   620 Proxy individuals    -   650 Appeal value    -   660 Weighting coefficient    -   670 Weighting coefficient    -   680 Previously observed individuals    -   690 Candidate individual    -   700 Appeal estimation device    -   705 Digital image capture device    -   710 Soft-copy display    -   720 Observer    -   730 Candidate individual

1. An appeal estimation system for estimating a personal appeal of acandidate individual to an observer comprising: a digital image capturedevice for capturing a digital image of the candidate individual; asoft-copy display; a data processing system; and a personal appeal modelmemory system communicatively connected to the data processing systemand storing information about a set of proxy individuals includingdigital images of each proxy individual and personal appeal values ofeach proxy individual to the observer; and a program memory systemcommunicatively connected to the data processing system and storinginstructions configured to cause the data processing system to implementa method for estimating the personal appeal of the candidate individualto the observer comprising: a) capturing one or more digital images ofthe candidate individual using the digital image capture device; b)determining image similarity metrics between the digital images for thecandidate individual and the digital images for each proxy individual;c) determining similarity values between the candidate individual andeach proxy individual responsive to the determined image similaritymetrics; d) estimating the personal appeal of the candidate individualto the observer by determining a weighted combination of the personalappeal values for the proxy individuals, wherein the weightedcombination uses weighting coefficients that are determined responsiveto the similarity values between the candidate individual and thecorresponding proxy individual; and e) displaying an indication of theestimated personal appeal on the soft-copy display.
 2. The appealestimation system of claim 1 wherein personal appeal values for at leasta subset of the proxy individuals are determined by having the observerevaluate the personal appeal of the proxy individuals.
 3. The appealestimation system of claim 2 wherein the evaluation of the personalappeal of the proxy individuals includes having the observer evaluatethe physical attractiveness of the proxy individuals based on viewingthe digital images of the proxy individuals.
 4. The appeal estimationsystem of claim 2 wherein the evaluation of the personal appeal of theproxy individuals includes having the observer evaluate non-imageinformation pertaining to the proxy individuals.
 5. The appealestimation system of claim 4 wherein the non-image information includesage, height, weight, hair color, hair style, income, activities, levelof education, personality, social affiliations or politicalaffiliations.
 6. The appeal estimation system of claim 1 whereinpersonal appeal values for at least a subset of the proxy individualsare determined by: (i) designating a set of proxy observers and a set oftraining individuals, wherein the subset of proxy individuals areincluded in the set of training individuals; (ii) determining trainingpersonal appeal values of the training individuals to the proxyobservers; (iii) determining observer personal appeal values of at leasta subset of the training individuals to the observer; (iv) determining aproxy observer model that relates the training personal appeal values tothe observer personal appeal values; and (v) using the proxy observermodel to determine the personal appeal values of the proxy individualsto the observer responsive to the training personal appeal values. 7.The appeal estimation system of claim 6 wherein the proxy observer modeldetermines the personal appeal value of a particular proxy individual tothe observer by determining a weighted combination of correspondingtraining personal appeal values of the particular proxy individual tothe proxy observers, and wherein the weighted combination uses weightingcoefficients determined responsive to the training personal appealvalues and the observer personal appeal values.
 8. The appeal estimationsystem of claim 7 wherein the weighted combination is a weightedsummation, and wherein the weighting coefficients are determined using alinear least squares regression algorithm.
 9. The appeal estimationsystem of claim 1 wherein the image similarity metric is a facialsimilarity metric.
 10. The appeal estimation system of claim 1 whereinthe similarity values are also determined responsive to the similarityof non-image information pertaining to the candidate individual and theproxy individuals.
 11. The appeal estimation system of claim 10 whereinthe non-image information includes age, height, weight, hair color, hairstyle, income, activities, level of education, personality, socialaffiliations or political affiliations.
 12. The appeal estimation systemof claim 1 wherein the weighted combination of the personal appealvalues is a weighted summation, and wherein the weighting coefficientfor a particular proxy individual is determined by normalizing thesimilarity values for the particular proxy individual by the sum of thesimilarity values for all of the proxy individuals.
 13. The method ofclaim 1 wherein the estimated personal appeal for a plurality ofcandidate individuals is simultaneously displayed on the user interface.14. The appeal estimation system of claim 13 wherein the candidateindividuals are ordered on the user interface according to the estimatedpersonal appeal.
 15. The appeal estimation system of claim 1 wherein theappeal estimation system includes a mobile computational device.
 16. Theappeal estimation system of claim 15 wherein the data processing systemincludes an internal data processor within the mobile computationaldevice, and wherein steps a)-e) are performed using the internal dataprocessor.
 17. The appeal estimation system of claim 15 wherein the dataprocessing system includes a remote data processor on a remote computingsystem, communicably-connected to the mobile computational device, andwherein steps b)-d) are performed on the remote data processor.
 18. Theappeal estimation system of claim 15 wherein the mobile computationaldevice is a cellular telephone or a tablet computer.
 19. A portableappeal estimation device for estimating a personal appeal of a candidateindividual to an observer comprising: a digital image capture module forcapturing a digital image of the candidate individual; a soft-copydisplay; a data processing system; and a personal appeal model memorysystem communicatively connected to the data processing system andstoring information about a set of proxy individuals including digitalimages of each proxy individual and personal appeal values of each proxyindividual to the observer; and a program memory system communicativelyconnected to the data processing system and storing instructionsconfigured to cause the data processing system to implement a method forestimating the personal appeal of the candidate individual to theobserver comprising: a) capturing one or more digital images of thecandidate individual using the digital image capture module; b)determining image similarity metrics between the digital images for thecandidate individual and the digital images for each proxy individual;c) determining similarity values between the candidate individual andeach proxy individual responsive to the determined image similaritymetrics; d) estimating the personal appeal of the candidate individualto the observer by determining a weighted combination of the personalappeal values for the proxy individuals, wherein the weightedcombination uses weighting coefficients that are determined responsiveto the similarity values between the candidate individual and thecorresponding proxy individual; and e) displaying an indication of theestimated personal appeal on the soft-copy display.